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FOREWORD 


One  mission  of  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  is  to  share  the  results  of  its  training  technology  research 
with  non-Department  of  Defense  (DoD)  Federal  agencies  and  with  the  civilian 
community.  This  mission  to  DoD  policy  on  technology  transfer  and  dual-use 
research.  The  policy  seeks  to  preserve  and  extend  military  technology  in  light  of 
ongoing  downsizing  and  budget  reductions. 

This  Research  Report  is  the  result  of  a  cooperative  effort  with  the  National 
Fire  Academy  (NFA)  in  Emmitsburg,  Maryland.  ARI  is  working  with  NFA  to 
apply  military  research  products,  experiences,  and  "lessons  learned"  to  the 
training  of  fire-groxmd  incident  commanders.  The  report  sximmarizes  an  initial 
effort  to  study  the  feasibility  and  functional  requirements  for  introducing 
computer-supported  simulation  to  train  fire-groimd  command  and  control. 

The  report  documents  a  number  of  shortfalls  in  managing  the  original, 
mechanically  based,  labor-intensive  simulation  methods  for  incident  command 
training.  It  establishes  a  clear  need  for  computer-based  upgrading.  The  report 
provides  specific  recommendations  in  support  of  NFA’s  short-range  goals  to 
upgrade  campus  training  and  long-range  goals  to  distribute  simulation-based 
training  nationwide. 

Preliminary  results  of  the  study  were  briefed  to  Acting  NFA  Superintendent 
James  F.  Coyle  in  February  1994,  and  used  to  help  plan  procurements  for  a  new 
simulation  facility  at  the  Emmitsburg  campus.  A  working  draft  of  the  report  is 
being  used  by  NFA  contractors  as  a  source  document. 


^^DGAR  M.  JOHNSON 
Director 
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COMPUTER-SUPPORTED  SIMULATION  AT  THE  NATIONAL  FIRE  ACADEMY: 
LESSONS  LEARNED  FOR  INCIDENT  COMMAND  TRAINING 


EXECUTIVE  SUMMARY 


Requirement: 

Determine  the  feasibility  of  and  functional  requirements  for  computer-supported 
simulation-based  training  of  fire-ground  incident  command  and  control. 


Procedure: 

Review  training  practices.  To  develop  a  baseline  of  information,  a  test-bed 
course  was  observed:  "Command  &  Control  of  Fire  Department  Major  Operations." 
A  log  of  activities  was  maintained  during  four  simulation  exercises.  The  purpose  of 
the  log  was  to  describe  simulation  methods  and  to  document  instructor/student 
activities  and  simulation  roles.  In  addition,  students  and  instructors  were  interviewed 
during  and  after  class  to  determine  their  reactions  to  the  training  methodology. 

Review  simulation  methods.  Using  the  data  discussed  in  the  previous 
paragraph  and  ancillary  reviews  of  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  research,  we  set  out  to  determine  how  the  management  and  use 
of  simulation-based  training  of  fire-ground  incident  command  might  be  upgraded  by 
computer  technology.  We  determined  ways  to  systematically  incorporate  military 
decision-making  research  into  the  design  of  simulation  exercises. 


Findings: 

The  baseline  data  revealed  that  instructors  are  required  to  play  10  conflicting 
and  rapidly  shifting  roles  at  up  to  14  simulation  sites  in  6  rooms  of  the  training  facility. 
These  conflicting  roles  severely  limit  training  effectiveness.  A  tabulation  of  these 
roles  and  sites  provides  a  starting  point  for  planning  and  evaluating  computer-based 
upgrades.  Such  upgrades  should  be  done  incrementally,  beginning  with  replacement 
of  the  mechanical  components  of  the  baseline  simulation  facilities.  Further,  a  training 
system  rather  than  simulation  perspective  should  be  the  foundation  for  an  upgrade 
strategy.  A  number  of  systems  issues  were  identified  for  consideration  in  any 
technology  upgrade.  These  include  'front-end'  preparation  of  the  trainees;  exercise 
conducted  and  management;  performance  assessment  and  post-incident  analysis;  and 


VII 


integration  into  the  training  exercises  of  various  perspectives  on  how  to  teach  decision 
making.  The  issues  were  detailed  and  recommendations  for  how  to  address  them  in 
planning  technology  upgrades  were  offered. 


Utilization  of  Findings: 

The  results  of  this  study  were  used  to  help  design  and  acquire  a  computer- 
supported  simulation  facility  for  training  fire-ground  incident  command  at  the  National 
Fire  Academy.  In  addition,  the  report  is  being  used  as  a  source  document  for  long- 
range  planning  of  future  upgrades  and  for  distributed  training. 


COMPUTER-SUPPORTED  SIMULATION  AT  THE  NATIONAL  FIRE  ACADEMY: 
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COMPUTER-SUPPORTED  SIMULATION  AT  THE  NATIONAL  FIRE  ACADEMY: 

LESSONS  LEARNED  FOR  INCIDENT  COMMAND  TRAINING 

INTRODUCTION 

This  report  fulfills  an  agreement  between  the  U.S.  Army  Research  Institute  (ARI) 
and  the  National  Fire  Academy  (NFA).  We  undertook  to  examine  the  need  and 
functional  requirements  for  computer-based  upgrading  of  training  simulation  at  the 
Emmitsburg  campus.  The  report  documents  opportunities  for  enhancing  management 
and  use  of  computer-supported  simulation  to  train  fire  scene  incident  commanders. 
But  it  also  looks  to  the  future  with  longer-range  recommendations  on  distributed 
training.  The  current  effort  is  part  of  a  broader  partnership  aimed  at  sharing  Army 
training  experiences  and  research  products  with  NFA. 

The  remainder  of  this  report  is  organized  as  follows:  a  Background  Section 
describes  the  NFA/ARI  partnership  and  its  rationale.  Why  have  the  agencies  joined 
forces?  The  next  section  describes  a  study  of  the  current  NFA  training  environment 
as  the  baseline  from  which  computer-supported  upgrades  will  be  implemented 
(beginning  in  November  1994).  Subsequent  sections  detail  lessons  learned  from  the 
study  and  recommendations  for  near  and  longer-term  developments.  This  paper 
includes  findings  and  recommendations  from  several  preliminary  documents  prepared 
between  January  and  October  1994  (Mirabella,  1994b;  Macpherson  &  Mirabella, 
1994;  Mirabella,  Satterfield,  &  Wood,  1994). 

BACKGROUND 
The  NFA/ARI  Partnership 

ARI  and  NFA  are  pursuing  a  joint  effort  to  transfer  training  and  training 
development  technology  from  the  U.S.  Army  to  the  Federal  Emergency  Management 
Agency  (FEMA).  The  partnership  is  responsive  to  the  President's  initiative  to  revitalize 
the  Federal  Government  (U.S.  Office  of  the  Vice  President,  1993).  A  task  force  led 
by  the  Vice  President  has  encouraged  federal  agencies  to  share  knowledge  and 
resources  among  themselves  and  with  the  private  sector.  The  Department  of  Defense 
(DOD)  actively  encourages  such  cooperation  (Department  of  Defense,  1 992;  Deutch, 
1994).  The  U.S.  Army's  participation  serves  its  growing  role  in  peacetime  emergency 
operations,  including  fire  fighting.  NFA  training  developments  may  provide  'lessons 
learned'  for  Army  applications. 

The  training  responsibility  of  FEMA  at  the  National  Emergency  Training  Center 
(NETC),  Emmitsburg,  Maryland,  provides  a  fruitful  opportunity  to  implement  the  Vice 
President's  challenge.  The  growing  frequency,  complexity,  and  devastation  of  fire  and 
other  emergencies  are  severely  taxing  our  nation's  ability  to  fully  train  the  total 
community  of  fire-ground  commanders.  To  meet  this  challenge,  NFA  has  initiated  a 
program  to  leverage  the  amount  and  effectiveness  of  simulation-based  training 
provided  by  its  campus  in  Emmitsburg,  MD. 
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Goals  of  the  Partnership 


NFA  is  revising  courses  in  the  command  and  control  (C&C)  of  fire  and  emergency 
service  operations  at  major  incidents  and  natural  disasters.  It  wants  to  computerize  its 
simulation-based  instruction  for  multi-threat,  multi-resource  management  environments. 
Moreover,  NFA  wants  to  incorporate  results  of  the  latest  scientific  research  on  expert 
decision  making  into  its  C&C  program.  Longer-range,  the  academy  seeks  to  develop 
affordable  ways  to  distribute  its  training  nation-wide,  through  distance  education  (DE) 
and  ultimately  through  distributed  interactive  simulation  (DIS)  technology.  Its  thrust 
towards  improved  training  technology  stems  from  a  study  conducted  by  panels  of 
emergency  management  experts  (W.  Satterfield,  personal  communication,  March,  1994). 

In  January  1994,  ARI  agreed  to  support  NFA  training  enhancement  goals  by 
sharing  Army  experiences  and  research  in  training  simulation  development.  The  rationale 
for  the  resulting  partnership  is  detailed  in  the  Appendix.  But  in  summary,  the  agencies 
share  common  training  development  problems,  ARI  has  transferable  experience  and 
products,  and  the  partnership  is  consistent  with  the  Army’s  growing  role  in  peacetime 
emergency  management. 


Purpose  of  the  Study 

As  an  initial  effort,  ARI  agreed  to  draw  on  its  experience  to  study  the  feasibility  of 
and  functional  requirements  for  upgrading  simulations  for  fire-ground  incident  command 
training.  The  remaining  sections  of  the  report  summarize  that  study.  The  study  was  one 
of  multiple  activities  carried  out  during  the  first  year  under  the  Partnership’s 
Memorandum  of  Agreement.  For  example,  ARI  provided  technical  advisory  service.  It 
assisted  NFA  with  planning  the  development  of  computer-based  instruction  for  fire-safety 
design  in  architecture.  It  provided  advice  on  how  to  incorporate  Army  concepts  of 
performance-oriented  instruction  into  the  fire-safety  design  courseware.  ARI  also 
conducted  a  preliminary  study  of  the  transfer  potential  of  ARI  research  products.  These 
efforts  will  be  referenced  in  the  Appendix. 
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METHODS 


Review  Current  Training  Practices 

The  purpose  of  the  review  was  to  generate  a  baseline  of  information.  This 
information  would  help  determine  how  best  to  incorporate  computer  technology  and 
research  on  decision  making  into  simulation  exercises.  The  course  "Command  &  Control 
of  Fire  Department  Major  Operations"  was  selected  as  a  testbed  for  observation  and 
analysis.  This  course  trains  company  officers  and  battalion  chiefs  to  command  several 
companies  on  the  fire  ground.  The  program  of  instruction  and  course  materials  were 
reviewed.  Subsequently,  the  senior  author  observed  the  course  and  maintained  an 
activity  log  for  the  simulation  exercises.  The  course  was  observed  using  an  adaptation 
of  methods  developed  in  an  earlier,  similar  ARI  study  (Ramsay,  Kessler,  &  Mirabella, 
1988).  The  primary  purpose  of  the  log  was  to  describe  simulation  methods,  and  to 
document  instructor/student  activities  and  simulation  roles. 

n 

Review  Simulation  in  the  Test-Bed  Course  on 
Major  Fire-Ground  Operations 

Four  simulation  exercises  were  observed  during  the  second  week  of  a  two-week 
course.  Each  exercise  included  a  start-up  brief  by  the  Chief  Instructor  (Cl),  the  exercise 
itself,  and  a  post-incident  analysis  (PIA).  The  senior  author  kept  an  approximate,  time- 
stamped  log  that  focused  on  the  Chief  and  Assistant  Instructor  activities.  The  senior 
author  also  informally  (i.e.,  off-the-record)  ’interviewed’  students  and  instructors  during 
and  after  class.  These  observations  provided  a  basis  for  analyzing  the  feasibility  of  and 
functional  requirements  for  establishing  a  Computer-Supported  Command  and  Control 
Simulation  Facility  at  NFA. 

A  follow-up  effort  was  then  made  to  identify  and  analyze,  instructional  design, 
software,  and  hardware  issues  to  be  considered  in  upgrading  the  baseline  incident 
command  simulation. 
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FINDINGS 


Description  of  the  Testbed  Course 

The  course  was  a  two-week  program  to  train  commanders  of  multi-alarm 
operations.  It  included  25  students  and  two  instructors.  Week  1  included  lectures  and 
class  drills  to  teach  the  incident  command  system  and  introduce  students  to  Rapid  Fire 
Ground  Decision  Making  (RFGM).  The  goal  of  Week  1  was  to  provide  students  with  a 
shared  understanding  of  a  ’doctrinal’  approach  to  planning  and  executing  incident 
command.  Week  2  began  with  a  day  of  lectures  and  continued  with  three  days  of  multi¬ 
alarm  simulation  exercises.  Friday  was  graduation  day.  Thus  the  ratio  of  lecture/drill  to 
simulation  was  2:1. 


Baseline  Simulation  Facilities  and  Exercise  Play^ 

Figure  1  shows  the  facilities  that  were  used  to  conduct  simulation  exercises  for 
the  testbed  course.  Students  initially  congregated  in  the  Classroom  Area  for  a  pre-brief 
on  the  up-coming  multi-alarm  exercise.  With  the  help  of  a  projected  photo  and  plan  view 
of  the  fire  site,  one  of  two  instructors  explained  the  exercise  problem,  e.g.,  construction 
of  the  building  or  buildings  on  fire;  street  layout  and  hydrant  locations.  The  instructor  also 
explained  the  strategies  and  tactics  that  might  be  used  and  the  problems  that  might  be 
encountered. 

Students  were  then  assigned  roles:  initial  incident  commander  (1C),  follow-up  1C 
(Battalion,  Assistant  District,  or  District  Chief),  tactical  commander  (engine  or  ladder 
company  chief),  company  crewman,  or  dispatcher.  Tactical  commanders  and  their  crews 
assembled  in  the  ’Fire  House’  Area;  dispatchers  went  to  the  Dispatcher  Station. 
Dispatchers  began  the  play  with  a  radio  call  to  the  initial  1C  (first  alarm  tactical 
commander). 

The  initial  1C  ’arrived’  on  the  scene  with  his  company  to  set  up  a  command  post 
and  begin  managing  the  incident.  He  saw  a  projection  of  a  photo  of  the  front  of  a  fire 
scene  with  its  fire-smoke  overlay.  As  he  saw  the  need  for  more  resources  he  issued 
additional  alarms.  Additional  companies  then  arrived  via  a  staging  area  with  a  higher- 
level  1C  (e.g..  Battalion  Chief)  who  would  take-over  incident  command. 

Simulation  Technology 

The  simulations  employed  slide  and  overhead  projectors  to  create  the  visual 
effects  of  a  fire-ground  scene.  Student-players  recorded  their  reactions  to  the  scene  on 
a  projected  plan  view  of  the  fire-ground.  For  example,  they  drew  hose  lines  or  vent  holes 
to  show  how  their  engine  or  truck  companies  were  carrying  out  the  incident 
commander’s  (1C)  assignments. 
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Figure  1.  Simulation  facilities  for  Major  Fire 
Operations  Course,  National  Fire  Academy, 
Emmitsburg,  MD. 


Instructors  observed  the  trainee  actions,  estimated  the  effects  of  those  actions, 
and  then  adjusted  a  fire/smoke  image  projected  on  a  ’burning’  structure  to  reflect  the 
trainees’  actions.  The  image  was  created  with  an  overhead  projector,  a  red  transparency 
sheet  on  the  projector’s  glass  plate,  and  sand  on  top  of  the  sheet.  The  instructor  created 
and  manipulated  an  opening  in  the  layer  of  sand  to  shape  the  fire  image. 

Simulation  Management 

Interviews  with  students  revealed  that  the  simulation  exercises  were  valuable  for 
developing  strategic  and  tactical  command  skills.  But  the  exercises  were  labor  intensive 
and  logistically  complicated.  The  two  instructors  were  required  to  play  ten  rapidly  shifting 
and  conflicting  roles  while  physically  moving  among  14  simulation  areas  (10  operational 
sites;  4  control  sites)  in  6  separate,  isolated  rooms  or  hallway  areas  (Table  1). 

Instructor  Roles 


1.  Stage  Director.  In  this  role  the  instructors  explained  how  to  play  the  game  - 
who  does  what,  when,  and  where.  The  role  began  with  a  lecture  describing  the 
simulation  and  its  operation.  It  continued  with  a  ’walk-through’  demonstration.  It 
was  required  through  out  the  simulation  week  at  all  the  simulation  sites. 

2.  Socratic  Coach.  As  coach  the  instructors  prompted  players  in  ’correct 
performance,  e.g.,  the  next  step  in  setting  up  the  command  post  or  carrying  out 
the  IC’s  assignment. 

3.  Observer/Evaiuator.  In  this  role,  the  instructor  had  to  keep  mental  notes  of 
reactions  of  23  students  at  approximately  9  operational  sites.  He  needed  these 
notes  for  the  PIA. 

4.  Tutor.  As  tutor,  the  instructor  explained  how  to  perform  various  tasks,  e.g.,  how 
a  ladder  company  should  carry  out  venting  operation. 

5.  Motivator.  The  instructors  made  concerted,  deliberate 
efforts  to  build  rapport,  and  encourage  and  support  students. 

6.  Simulation  Technician.  This  is  a  triple  role:  detect  student  responses,  interpret 
response  impact  on  the  fire  scene,  and  then  manipulate  the  visual  display  to 
reflect  student  tactics.  Observations  and  interviews  suggested  that  this  latter  role 
suffers  most  from  the  time-sharing  and  sampling  of  roles.  This  in  turn  limits  the 
amount  and  timeliness  of  feedback  to  students  and,  therefore,  training 
effectiveness. 

7.  Coordinator.  The  instructors  conferred  with  each  other  periodically  to  discuss 
exercise  control  issues  and  fire  scene  changes. 
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8.  Post-Incident  Analysis  Manager.  This  role  is  not  directly  played  during  the 
exercises.  But  clearly  the  exercise  technology,  management,  and  conduct  affect 
how  well  the  PIA  is  carried  out.  The  PIA,  in  turn,  is  a  critical  component  and  a 
limiting  factor  in  training  effectiveness. 

Simulation  Control  Sites: 


1.  Projection  room.  The  projection  room  contained  projection  systems  for 
displaying  the  front  and  rear  fire  scenes  respectively,  into  rooms  on  either  side  of 
it. 

2.  Plan  view  projections.  The  front  and  rear  scene  rooms  contained  a  plan  view 
projection  operated  by  a  student.  With  help  of  the  operator,  players  indicated  their 
responses  by  ’grease  penciling’  on  a  plastic  overlay  of  the  plan  view  of  the  fire 
scene. 

3.  Fire  House  Area.  Companies  assembled  here,  waiting  to  be  called. 

4.  Dispatcher  station.  One  student  was  assigned  to  dispatch  engine  or  ladder 
companies  from  the  Fire  House  Are  to  the  Staging  Area  or  from  the  Staging  Area 
to  fire  scenes.  He  was  seated  in  front  of  a  dispatcher’s  simulator  console,  though 
this  was  not  operational  at  the  time.  He  sent  his  dispatches  by  portable  radio. 

5.  Command  Station.  A  table  before  the  front  fire  scene  projection  simulated  the 
incident  command  station.  The  IC  would  set  up  a  status  sheet  to  track  companies 
and  their  assignments. 

6.  Staging  Area.  This  simulated  a  location  within  several 

blocks  of  the  fire  where  reserve  units  assembled,  waiting  to  be  called  by 
the  IC. 

7.  Company  Operation  Sites.  Engine  and  ladder  crews  or  sections  located 
themselves  before  the  front  or  rear  scene  projections  to  ’carry  out’  the  IC’s 
assignment,  e.g.  search  and  rescue,  ventilation,  fire  suppression. 

How  to  Use  the  Test-Bed  Descriptive  Data 

A  summary  of  instructor  roles  and  locations  is  presented  in  Table  1 .  The  table  can 
be  used  to  help  document  the  need  for  training  technology  upgrades  and  impartially 
support  an  audit  trail.  It  also  provides  a  baseline  to  help  guide  and  assess  training 
upgrades. 
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Table  1.  Instructor  Roles  and  Operational  Sites 


Instructor  Roles 

Simulation  Sites 

1 .  ’Stage  Director’  (explains  game  moves) 

1.  Projection  room:  front  view 

2.  ’Socratic  Coach’  (prompts  correct  behavior) 

2.  Projection  room:  rear  view 

3.  ’Observer/Evaluator’  (notes  performance  for 
PIA) 

3.  Plan  view  projection:  front 
scene  room 

4.  ’Tutor’  (explains  fire-fighting  procedures) 

4.  Plan  view  projection:  rear 
scene  room 

5.  ’Motivator’  (builds  rapport,  encourages, 
supports  students) 

5.  Fire  House  Area 

6.  ’Detector  of  student  responses 
(detects  actions  which  change  fire  scene) 

6.  Dispatcher  station 

7.  ’Assessor  of  response  impact’  (estimates 
effects  of  student  actions  on  fire  scene) 

7.  Command  station 

8.  ’Manipulator  of  visual  display’  (changes  fire 
scene  display  to  reflect  student  actions) 

8.  Staging  Area 

9.  ’Coordinator  (conferred  with  other  instructor 
on  exercise  control  and  fire-scene  displays) 

9.  Company  operation  sites 
(usually  3  to  6) 

10.  PIA  manager  (plans/chairs  the  PIA) 
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LESSONS  LEARNED  AND  RECOMMENDATIONS 


General  Conclusions  Regarding  Need  and  Feasibility 


Need 


Descriptive  data  on  simulation  management  and  operation  clearly  document  the 
need  for  computer  technology  enhancements.  They  reinforce  earlier  findings  by  expert 
panels  (W.  Satterfield,  personal  communication,  March,  1994).  Instructors  were  severely 
overloaded  in  managing  the  testbed  course  simulation  exercises:  10  conflicting  and 
shifting  roles  at  up  to  14  simulation  sites  in  6  rooms.  These  data  can  provide  a  partial 
baseline  for  assessing  training  upgrades. 

The  roles  of  simulation  technician  are  evident  starting  points  for  computer-aiding. 
Since  the  changes  in  fire  scene  are  relatively  few,  not  complicated,  and  discreet,  a 
library  of  flame/smoke  displays  (e.g.,  on  Computer  Disk  [CD]  Read  Only  Memory  [ROM], 
is  a  candidate  for  calling  up  and  displaying  successive  decreases  or  increases  in  flames 
and  smoke  superimposed  on  a  target  building.  A  longer  range  candidate  is  animated, 
interactive  computer  generated  imagery  model  of  fire-smoke  behavior. 

Simulation  upgrade  at  NFA  is  a  critical  first  step  in  the  longer-range  goal  of 
distributing  1C  training  nationwide.  The  need  for  this  latter  goal  is  clear:  Training 
development  and  delivery  at  NFA  are  extraordinarily  well  executed  in  spite  of  the 
obsolete  test-bed  technology  which  has  been  available.  Staff  developers  and  instructors 
are  exceptionally  competent.  But  the  net  value  of  that  training  is  severely  limited.  The 
NFA  campus  can  accommodate  only  a  small  fraction  of  the  Emergency  Management 
(EM)  population  in  need  of  1C  training. 

Feasibility 

An  affordable  and  technically  feasible  strategy  for  upgrading  incident  command 
training  requires  two  features.  First,  it  should  be  incremental.  Step  1  should  be  to 
replace  the  mechanical  components  of  the  testbed  system  with  computer 
components.  Budget  constraints  and  the  nature  of  the  training  and  requirements 
suggest  that  a  fully  automated  interactive  simulation  facility  is  not  feasible  as  a  first 
step,  and  not  cost-effective.  A  model  for  a  'first  step'  facility  is  provided  by  the  1C 
training  system  of  the  London  Police  Department  (Crego  &  Powell,  1 994) .  The  London 
Fire  Department  has  determined  that  this  system  is  also  suitable  for  its  own  1C  rating. 

Beyond  cost,  lie  additional  reasons  for  an  incremental  approach.  Many  training 
system  issues  need  to  be  addressed  in  upgrading  simulation  hardware/software,  e.g., 
proper  mixture  of  simulation  exercises  with  other  course  components;  ’front  end’ 
preparation  for  the  exercises;  and,  student  response  monitoring,  recording,  and  display 
for  PIAs.  It’s  easier  to  address  such  issues  with  modest  increments  in  technology. 
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An  incremental  approach  also  makes  it  easier  to  assess  the  value-added  by 
training  technology  upgrades.  We  recommend  that  NFA  include  such  an  assessment  in 
its  planning  for  upgrades.  A  systematic  assessment  of  value-added  would  encourage 
support  for  further  upgrades,  provide  a  baseline  for  assessing  their  impact,  and  yield 
’consumer’  suggestions  for  ’product  improvement’.  ARI’s  assessment  of  the  Multi-Service 
Distributed  Training  Test  Bed  (MDT2)  has  resulted  in  each  of  the  foregoing  benefits.  The 
Army’s  experience  with  assessing  ’value-added’  for  the  MDT2  system  may  have  some 
transferable  lessons  for  NFA  training  developments  (Mirabella,  1994a). 

In  addition  to  an  incremental  approach,  a  feasible  strategy  requires  a  training 
system  perspective  rather  than  a  simulation  perspective.  Here  a  lesson  can  be  learned 
from  the  military.  The  Army  developed  the  Simulation  Networking  (SIMNET)  System  as 
simulator  rather  than  as  a  training  system.  For  example,  it  did  not  incorporate  a 
measurement/data  collection  capability  to  support  after-action  reviews  and  training 
development.  Yet,  the  critical  role  of  effective  feedback  in  successful  unit  training 
simulation  is  well  established  (e.g.,  Thomas,  Barber,  &  Kaplan,  1989).  We  will  say  more 
about  the  implications  of  a  systems  view  and  provide  specific  recommendations  in 
following  sections  of  the  report. 
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Lessons  Learned/Recommendations  — Near  Term 


In  keeping  with  the  foregoing  comments,  the  following  lessons  learned  and 
recommendations  will  focus  on  simulation  issues,  but  will  do  so  from  a  training 
systems  perspective. 

'Front-End  Preparation' 

Lessons  Learned; 

Consider  improving  and  better  integrating  the  'front-end'  training  with  simulation 
exercises.  For  example,  the  student  manual  chapter  on  the  Incident  Command  System 
(ICS)  introduces  ICS  at  the  highest  level  of  abstraction  (Command  and  Control  of  Fire 
Department  Major  Operations  Student  Manual,  Page  2-19).  Only  briefly  and  at  the  end 
of  the  chapter  does  the  manual  relate  ICS  to  course  level  operations.  This  strategy  of 
general  and  abstract  to  specific  and  concrete  is  inconsistent  with  research  finds  that 
recommend  the  reverse  strategy  (Mirabella,  Macpherson,  &  Patterson,  1989). 

'Front-end'  preparation  (in  the  test-bed  course)  consumed  6  out  of  9  days,  with 
3  days  devoted  to  simulation  exercises  (SIMEXs).  This  again  is  inconsistent  with 
research  showing  that  'front-end'  activity  should  not  exceed  50%  of  a  training  cycle 
(Mirabella  et  al.,  1989). 

A  mechanisms  to  help  link  the  'front-end'  preparation  to  SIMEXs  would  be  an 
easy  to  use,  hard-copy,  decision  aid.  Candidates  for  decision  aid  formatting  are  the 
Unit  II  Appendix  material  on  1C  staffing  (Participant  Manual  for  Major  Operations 
Course);  a  concise  summary  of  ICS  principles  applied  to  multi-alarm  scene 
management,  with  practical  examples;  basic  skill  information  such  as  how  to  lay  out 
a  Resource  and  Situation  Status  Record.  One  model  for  such  a  decision  aid  is  ARI's 
"Commander's  Battle  Staff  Handbook"  (U.S.  Army  Research  Institute,  1993). 

Recommendations: 

Given  the  short  amount  of  course  time,  focus  ICS  immediately  at  the  battalion 
operations  level  for  which  students  are  training.  Alert  students  to  other  levels  of 
ICS  application,  but  don't  dwell  on  them.  In  other  words,  reverse  the  emphasis 
in  current  training  of  ICS.  Make  treatment  of  ICS  less  abstract,  more  concrete. 

Tie  the  ICS  instruction  explicitly  into  training  objectives  and  the  exercises  to  be 
run  in  the  SIMEXs.  Begin  the  SIMEXs  by  Day  3.  Students  prefer  more  simulation, 
less  traditional  classroom  activity.  This  change  will  double  the  amount  of 
practice  for  each  student  in  the  1C  role.  Time  on  task  still  is  the  most  reliable  and 
potent  variable  in  training  effectiveness  (Mirabella  et  al.,  1989). 
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Develop  a  pocket  book  of  concise  information  for  use  in  the  SIMEXs  and  as  part 
of  a  take-home  package  which  students  can  share  with  colleagues  at  home 
station. 

In  planning  and  implementing  computer-based  upgrades,  consider  ways  to  use 
those  upgrades  to  support  pre-exercise  briefing  and  orientation. 

SIMEX  Conduct  and  Management 

Lessons  Learned 

We  summarized  in  Table  1  what  we  learned  about  problems  in  exercise 
management  and  control.  The  table  identifies  10  instructor  roles,  played  at  up  to  14 
instructional  sites,  in  6  rooms.  It  was  clear  in  many  cases  that  these  conflicting  roles 
limited  the  training  capabilities  of  the  simulation  system.  For  example,  fire/smoke 
displays  were  changed  infrequently  when  instructors  were  extra  busy  coaching  students 
or  directing  game  activities.  When  this  happened  student  performance  and  its 
consequences  were  ’out  of  synch’.  Fire  and  smoke  cues  could  not,  under  these 
circumstances,  provide  the  feedback  deemed  essential  by  instructional  theory. 
Furthermore,  performance  monitoring  by  instructors,  necessarily,  was  incomplete. 

Recommendations 

Use  Table  1  as  a  baseline  checklist  in  planning,  implementing,  and  evaluating 
incremental  upgrades  in  1C  training  technology. 

As  part  of  the  development  of  the  library  of  fire  scenes  for  upgraded  simulation 
facilities,  document  critical  decisions  to  be  made  in  the  SIMEXs  to  insure  that  the 
library  contains  an  adequate  range  and  variety  of  related,  critical  cues.  This 
documentation  would  be  part  of  a  broader  process  of  integrating  Klein’s 
Recognition-Primed  Decision  Making  (RPD)  methodology  into  the  design  of 
SIMEX  scenarios.  (See  Page  17). 

Computer  Support  for  Measurement  and  PIA 

Lesson  Learned:  Computer-aiding  should  support  not  just  the  display  of  the  fire 
scenes  and  exercise  control,  but  performance  measurement,  and  the  related  PIA. 
Computerizing  the  fire  scene  display  will  not  be  useful  if  is  not  made  for  monitoring  and 
recording  student  responses  which  would  change  the  fire  scene  in  actual  "combat’  and 
provide  data  for  the  PIA. 

An  incremental  solution  to  tracking  student  behavior  that  triggers  fire  scene 
changes  is  partial  interactivity.  In  the  test-  bed  course  students  grease-penciled  their 
responses  on  a  projected  plan  view  at  two  facilitator  stations.  In  concept,  the  mechanical 
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overhead  can  be  replaced  by  an  electronic  pad  and  computerized  projection  with  time- 
stamped  entries  showing  truck  and  hose-line  movements.  These  views  would  be  callable 
from  the  instructor  stations.  The  time-stamped  views,  could  later  support  the  PIA.  With 
two  monitors  at  his  work  station,  an  instructor  could  track  both  the  fire  scene  and  student 
performance  from  the  same  location. 

Recommendations: 

Examine  use  of  the  electronic  white  board  to  record  and  ’play  back’  student 
responses  for  PIAs.  This  technology  would  upgrade  the  plan  view  projection  used 
in  the  baseline  multi-alarm  course.  It  would  permit  students  to  continue  to  ’grease 
pencil’  their  responses,  e.g.,  drag  hose  line  to  the  rear  of  a  building.  But  it  would 
record  responses  electronically  for  display  at  networked  monitors.  With  some 
method  of  time-tagging,  the  responses  could  be  recalled  in  sequence  during  the 
PIA  for  discussion  and  diagnosis.  The  technology  to  do  this  is  off-the-shelf  and 
inexpensive,  e.g.,  ’SOFTBOARD’,  ("Algebra  Telecourse",  1994). 

Audio  recordings:  Examine  the  use  of  audio  recordings  to  assess  and  diagnose 
communication  and  coordination  problems  during  IC  exercises.  Minimally  a 
method  for  time-tagging  needs  to  be  developed.  Otherwise,  audio  data  would  be 
inaccessible  for  timely  use  in  PIAs. 

The  challenge  for  audio  and  visual  display  recording  and  use  is  to  retrieve  the 
data  and  format  it  quickly  in  a  way  that  trainers  and  students  find  useful.  A  lesson 
to  be  learned  from  Army  experience  is  that  this  challenge  should  be  addressed 
early  in  simulation  development,  especially  for  locally  networked  and  distributed 
simulation.  We  recommend  that  NFA  address  this  challenge  as  part  of  its  current 
training  upgrade  and  in  planning  further  evolutions  of  training  simulation. 

Integration  of  Decision  Making  Perspectives  Into  SIMEXs 

Part  of  the  systems  approach  is  to  decide  how  to  incorporate  multiple  perspectives 
on  decision  making  into  the  SIMEXs.  Table  2  identifies  perspectives  shown  by  research 
to  be  critical  to  effective  decision  making  training  (Mirabella,  1994b). 
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Table  2.  Perspectives  in  Training  Decision  Making  (DM)  for  Dangerous/Emergency  Situations 
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Lessons  Learned:  Case  Based  Reasoning 

The  first  condition  in  Table  2  is  labeled  Case-Based  Reasoning.  This  is 
synonymous  with  reasoning  by  analogy  which  is  the  basis  for  Klein’s  Recognition  Primed 
Decision  Making  (RPD)  or  alternatively,  Rapid  Fire  Ground  Decision  Making  (RFGDM). 
Klein  showed  that  for  a  wide  variety  of  dangerous,  stressful  situations,  including  fire- 
ground  command,  RPD  (RFGDM)  is  the  primary  way  that  commanders  make  decision 
(Klein,  1986,  1989,  1993;  Klein,  Calderwood,  Clinton-cirocco,  1988;  Klein,  Zsambok,  & 
Thordsen,  1993).  From  past  experience  they  recognize  critical  cues  and  respond  in  ways 
that  have  resulted  in  acceptable  outcomes. 

But  some  caveats  constrain  the  use  of  RPD  as  a  training  panacea  -  a  "silver 
bullet."  First,  the  method  can  result  in  ineffective  performance  (Klein,  1993).  Secondly, 
some  ways  of  teaching  analogical  reasoning  are  better  than  others.  How  it’s  taught 
makes  a  difference  (Cohen,  1994;  Kellman  &  Kaiser,  1994;  Walker,  Fisk,  Phipps,  &  Kirlik, 
1994).  Third,  its  validity  for  1C  training,  while  plausible  has  not  been  established.  Clearly 
it  describes  the  typical  behavior  of  expert  ICs  in  operational  settings.  But  students  at 
NFA  are  not  1C  experts  and  the  multi-alarm  course  is  not  an  operational  setting. 

Recommendations: 

We  agree  with  Klein’s  approach  of  focusing  on  critical  decisions  first  (Klein,  1994) 
and  then  defining  critical  cues.  We  would  take  this  a  step  further,  however,  and 
integrate  RFGDM  with  instruction  in  ICS.  Secondly,  we  would  use  scenario  design 
to  address  limitations  in  the  RPD  method,  e.g.,  introduce  realistic  "red  herrings". 

Lessons  Learned:  Shared  Understanding  (’Mental  Models’) 

A  growing  body  of  research  indicates  that  ’teams’  perform  more  effectively  (make 
better  decisions)  when  team  members  share  a  common  understanding  (’shared  mental 
model’)  about  each  others  roles,  the  job  environment,  and  tasks  to  be  performed.  The 
EM  community  discovered  and  implemented  this  principle  many  years  ago  when  it 
established  the  Incident  Command  System,  following  devastating  fires  in  southern 
California  (National  Fire  Academy,  1993).  The  NFA  appropriately  recognizes  the 
importance  of  ’shared  understandings’  (’shared  mental  models’)  by  featuring  ICS 
prominently  in  its  multi-alarm  command  program. 

Recommendations: 

NFA  can  increase  the  impact  of  ICS  training  and  the  shared  understanding  it 
promotes  by  integrating  it  more  tightly  and  more  concretely  with  other  aspects  of 
multi-alarm  training,  e.g.,  with  RFGDM  and  SlMEXs  (Mirabella  et  al.,  1989). 

See  earlier  paragraph  on  "Front-end"  preparation.  This  contrasts  with  teaching 
ICS  as  a  separate  introductory  module  in  Week  1. 
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Assess  applications  of  ARI  research  on  measurement  of  shared  understanding 
by  teams  (Perez,  1994a,  1994b).  This  body  of  work  may  yield  an  instrument  for 
measuring  the  impact  of  IC  simulation-based  training  upgrades  on  a  shared 
understanding  of  fire-scene  management. 

Lessons  Learned:  Metacognition 

The  IC  who  does  a  critically  incomplete  size-up,  but  thinks  he’s  done  an  adequate 
job  has  poor  metacognition.  He  has  poor  understanding  about  his  understanding  of  the 
fire-ground  situation. 

Consequence:  the  IC  may  fail  to  seek  additional,  required  information.  On  the  other 
hand,  an  IC  may  have  all  the  information  he  needs,  but  may  believe  otherwise. 
Consequence:  delays  in  decision  making. 

Poor  metacognition  can  interact  with  poor  use  of  RPD  techniques  to  result  in  poor 
if  not  disastrous  decisions.  An  IC  may  believe  he’s  ’read’  all  the  cues,  make  a  decision, 
and  then  discover  too  late  an  overlooked  piece  of  information. 

Metacognitive  analysis  refers  to  a  set  of  techniques  that  can  be  used  in  training 
and  on-the-job  to  monitor  situation  awareness  and  reduce  the  impact  of  inaccurate 
metacognition  (Cohen,  1994;  Cohen,  Adelman,  Tolcott,  Bresnick,  &  Marvin,  1994). 

Recommendation:  Research  sponsored  by  the  Army  and  the  Navy  identifies 
metacognitive  analysis  techniques  that  may  be  transferable  to  NFA  courses. 
Recommend  that  these  techniques  be  reviewed  for  their  applicability  to  SIMEXs 
at  NFA. 

Lessons  Learned:  "By  the  Numbers"  Basic  Skills. 

A  common  lesson  was  learned  from  the  MDT2  program  and  from  observation  of 
SIMEXs  in  the  testbest  multi-alarm  course.  Deficiencies  in  basic,  assumed  prerequisite 
skills  and  knowledge  can  reduce  the  efficiency  of  advanced  command  and  control 
training. 

For  example,  in  the  multi-alarm  course,  an  instructor,  on  one  occasion,  had  to  halt 
an  exercise,  to  explain  to  an  IC  the  basic  procedure  of  setting  up  a  status  sheet.  And 
during  a  PIA,  an  instructor  explained  at  length  the  advantages  of  divisional  vs  functional 
organization.  But  he  had  to  backtrack  when  it  became  evident  that  most  of  the  students 
did  not  understand  the  definitions  of  those  terms.  The  terms  were  explained  in  detail  in 
the  student’s  manual.  The  students  either  missed  the  explanations  or  were  not  able  to 
transfer  them  to  the  simulations.  This  misunderstanding  by  the  students  is  mentioned 
here  to  illustrate  the  need  to  closely  couple  the  ’front-end’  preparation,  i.e.,  ICS 
instruction,  with  the  SIMEXs. 
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Recommendation: 


Identify  and  log  missing  pre-requisites  during  the  simulations.  Incorporate 
remedial  instruction  for  the  most  critical  of  these  prerequisites  into  the  ’front-end’ 
preparation. 

Lesson  Learned:  Effective  Communication 

Research  has  shown  that  effective  communication  is  a  key  characteristic  of  teams 
that  perform  well  (e.g.,  Urban,  Bowers,  Monday,  &  Morgan,  1993).  Faulty  communication 
was  frequently  in  evidence  during  the  SIMEXs  observed.  This  included  tactical 
commander’s  not  providing  timely  updates  as  well  as  ICs  not  eliciting  such  updates. 

Recommendation: 

Identify  and  document  sources  and  categories  of  mis-  communication  during  the 
SIMEXs.  Develop  instruction  in  communication  techniques.  Incorporate  these  into 
’front-end’  preparation  and  integrate  with  SIMEXs  and  PIAs. 

Training  System  Assessment 


Lesson  Learned 

Though  student  reactions  to  courses  are  routinely  surveyed,  NFA  would  benefit 
from  a  more  systematic  approach  to  training  system  assessment.  Such  assessment  has 
materially  contributed  to  continuing  support  and  funding  for  the  MDT2  inter-service 
program,  engaged  the  early  and  constructive  involvement  of  prospective  MDT2  users, 
and  provided  invaluable  ideas  for  further  training  system  developments.  ARI  has 
conducted  a  considerable  amount  of  research  on  assessment  methodology  which  can 
be  adapted  for  use  at  NFA. 

Recommendation 

Examine  the  applicability  of  ARI  research  on  collective,  networked  training  system 
assessment.  A  framework  and  methodology  for  such  assessment  has  been 
developed  for  the  MDT2  program  (Mirabella,  1994a).  ARI  has  produced  an 
extensive  literature  on  methods  for  assessing  training  system  value  and 
characteristics  (e.g..  Babbit  &  Nystrom,  1989a,  1989b,  Boldovici  &  Bessemer, 
1994;  Burnside,  1990;  Burnside  &  Bessemer,  1991;  Riedel  etal.,  1994;  Shlechter 
et  al.,  1994;  Turnage,  Houser,  &  Hofmann,  1990). 
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Lessons  Learned/Recommendations  — Longer  Term 

The  purpose  of  this  section  is  to  provide  some  ideas  in  support  of  NFA's  future 
vision  and  general  roadmap  for  improving  and  expanding  simulation-based  1C  training. 
A  general  recommendation  is  to  proceed  incrementally  through  the  following  NFA 
identified  stages.  Use  Table  1  or  other  baseline  as  a  checklist  for  evaluating  each 
increment.® 

Simulation  Technology  Upgrades  at  the  NFA 

The  initial  upgrade  (in  progress  at  the  time  of  this  report)  will  permit  the 
instructors  to  select  fire  scenes  from  a  library  (e.g.,  CD-ROM).  A  step  considerably 
beyond  this  is  the  computer-generated  fire  scene  that  changes  continuously  in 
response  to  the  actions  of  tactical  commanders  and  crews,  as  well  as  to  non-human 
effects,  e.g.,  composition  of  structures  and  wind/draft  conditions.  The  images  would 
be  accurate  in  color,  pattern,  and  movement  for  multiple  perspectives  (front,  side, 
rear).  Further  fidelity  would  require  that  building  structures  respond  to  the  effects  of 
fire  and  smoke,  e.g.,  collapsing  walls.  Ultimate  realism  would  be  provided  by 
photographic  quality,  virtual  reality  simulation  (VRS). 

Such  full  interactivity  and  fidelity  is  not  currently  available.  But  a  primitive 
version  of  the  foregoing  technology  is  available.  It  takes  the  form  of  a  fire  and  smoke 
simulation  model  in  a  virtual  reality  program  for  fire-safe  building  design  and  fire 
training  fire  fighting  (Dawn,  1 994).  The  program  permits  the  user  to  navigate  through 
realistically  simulated  fire  and  smoke  in  a  burning  building.  The  user  could  be  either 
a  'trapped  victim'  or  a  'fire  fighter.'  Furthermore,  research  in  fire  simulation  is  in 
progress  in  Sweden  (Johan  Sellstroem,  Personal  Communication)  and  in  the  UK  (Tim 
Martin,  Personal  Communication). 

A  key  instructional  design  issue  is  how  much  fidelity  and  interactivity  are 
required  for  effective  training  of  ICs.  Army  experience  suggests  that  Command  Post 
Exercise  (CPX)  training  in  garrison  is  incomplete  until  units  can  implement  plans  in 
Field  Training  Exercises  (FTX).  The  reason  for  this  is  that  CPXs  depend  on  simulated 
intelligence  data  and  estimated  or  computed  effects  of  implementing  command  and 
control  decisions.  These  simulations  and  estimations  are  not  realistic  enough  to 
provide  all  the  training  necessary  to  the  command  staffs.  Current  NFA  multi-alarm 
simulation  is  more  than  a  CPX,  but  less  than  an  FTX.  Real  people,  not  computers  play 
the  roles  of  tactical  commanders  and  engine  or  ladder  crews  which  are  subordinate 
to  ICs.  But  if  the  subordinates  are  behaving  unrealistically,  they  will  generate 
unrealistic  cues  for  the  1C  to  act  on.  The  1C  is  likely  to  develop  unrealistic  or  otherwise 
inappropriate  command  habits. 
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The  other  side  of  the  coin  is  reflected  in  some  of  the  MDT2  assessment  data. 
Participants  in  the  multi-service  demonstration  indicated  that  they  found  the  MDT2 
environment  credible,  even  though  displays  were  cartoon-like  and  the  environment  was 
not  full-fidelity.  They  did  not  demand  full  photographic  fidelity.  And  they  expressed 
willingness  to  work  around  limitations  in  the  simulation.  A  similar  analysis  and 
assessment  of  NFA  student  needs  would  help  answer  the  question  about  how  much 
fidelity  is  useful  for  effective  training.  This  analysis  needs  to  be  carried  out  in  conjunction 
with  the  development  and  introduction  of  RPD  (RFGDM).  That  technology  requires  that 
critical  fire-ground  cues  be  identified  and  simulated  with  a  sufficient  degree  of  fidelity  to 
evoke  effective  responses  from  the  students. 

Recommendations: 

Catalog  a  range  of  technologies  to  support  incremental  increases  in 
fidelity  and  interactivity  and  define  the  status  of  the  required  technology,  e.g.,  off- 
shelf,  available  but  needs  to  be  adapted,  emerging  (e.g.,  under  research  and 
development).  Determine  how  such  methodology  can  be  added  to  the  initial 
upgrade.  The  initial  upgrade  with  computers  replacing  mechanical  components  as 
multi-media  gaming  aids  is  one  end  of  the  simulation  continuum.  The  outermost 
anchor  point  might  be  represented  by  students  wearing  virtual  reality  helmets, 
emersed  in  an  emergency  scene,  with  near  photographic  resolution,  interacting 
realistically  with  fire  and  smoke  which  in  turn  are  interacting  realistically  with 
combustibles  and  environmental  variables.  Between  these  anchor  points,  identify 
a  series  of  increasingly  interactive  technologies.  The  catalog  would  provide  a 
useful  set  of  ’road  marker’s’  in  planning  future  simulation  upgrades. 

Systematically  analyze  fidelity  requirements.  We  recommend  two  methods:  The 
first  is  to  trace  backward  in  the  chain  of  causal  events  from  critical  1C  decisions 
to  related  cues;  then  to  the  actions  of  tactical  commanders  and  crew;  finally,  to 
cues  preceding  those  actions.  This  is  a  variation  on  the  strategy  being  used  to 
introduce  RPD  into  the  Managing  Company  Tactical  Operations  (MCTO)  and  Fire 
Command  Operations  (FCO)  courses.  But  here  the  analysis  is  carried  back  a 
step.  The  objective  is  to  identify  the  essential  level  of  SIMEX  interactivity  and 
fidelity  which  subordinate  players  require  in  order  to  perform  in  sufficiently  realistic 
ways  to  provide  the  commander  with  realistic  cues  for  his  or  her  ’rapid  fire  ground 
decision  making.  The  second  method  is  to  survey  student  reactions  concerning 
credibility,  fidelity,  and  level  of  interactivity.  ARI  has  used  survey  techniques 
extensively  in  assessing  the  value  of  such  simulation  characteristics. 

Networking  on  Campus  to  the  Emergency  Management  Institute 

In  planning  for  linkages  to  the  Emergency  Management  Institute  (EMI),  consider 
that  this  is  the  equivalent  of  going  ’multi-service’.  Two  lessons  can  be  offered  from  MDT2 
experience: 
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a.  Consider  carefully  the  prerequisite,  ’single  service’  skills  and  the  need  for 
orientation  in  other  service  standard  operating  procedures  in  establishing  ’multi-service’ 
training. 

b.  ’Multi-service’  training  requires  a  top-down  commitment. 

Otherwise  conflicting  schedules,  goals,  and  related  needs  may  prove  difficult,  if  not 
insurmountable.  The  MDT2  project  team  worked  and  continues  to  work  around  such 
conflicts.  But  MDT2  is  a  prototype.  A  production  system  is  unlikely  to  work  without  top- 
down  direction. 

Networking  to  Regional  Centers  and  Fire  Service  Home  Stations 

A  range  of  network  models  are  available.  These  are  anchored  by  Distributed 
Learning  (DL)  at  the  "low"  technology  end  of  the  continuum  and  Distributed  Interactive 
Simulation  (DIS)  at  the  ’high’  end.  Each  has  a  different  set  of  instructional  purposes, 
design  and  technical  problems.  DL  as  used  in  the  literature  refers  primarily  to  distributed 
classroom  instruction.  DIS  refers  to  distributed  simulators  using  equivalent  data  bases, 
e.g.,  tanks  at  Ft  Knox  and  aircraft  at  Armstrong  Laboratory  on  or  over  the  same  terrain. 
The  existing  FEMA  Satellite  and  down  links  should  support  DL  and  the  initially  upgraded 
facilities.  The  Defense  Simulation  Internet  (DSI),  used  in  the  MDT2  Project  may  not  be 
a  suitable  technology  for  NFA  purposes.  It’s  expensive  and  unreliable. 

Consider  the  following  planning  issues: 

a.  Customer  Needs.  Consider  potential  military  as  well  as  civilian  customers, 
’created’  as  well  as  stated  needs;  and  realistic  constraints  on  potential  student  time.  The 
National  Guard  has  considerable  experience  with  addressing  this  latter  issue.  And  ARI, 
in  working  with  the  NG,  has  developed  supporting  methodology  (Hahn,  H.A.  Harbour, 
J.L.,  Wells,  R.A.,  Schurman,  D.L.,  &  Daveline,  K.A.,  1990;  Harbour,  Daveline,  Keith, 
Wells,  Shurman,  &  Hahn,  1990).  This  methodology,  for  example,  allows  soldiers  to  dial 
up  courseware  from  home  computers. 

b.  Type  of  Service.  Dial-in  self  vs.  group  instruction  with  an  instructor  on  one  or 
both  ends  -  asynchronous  vs  synchronous  (Hahn  et  al.  1990;  Harbour  et  al.,  1990). 

c.  Training  Cadre  Support.  The  cadre  needs  on  the  receiving  as  well  as  the 
sending  end  of  distributed  instruction  have  to  be  considered,  particularly  for  distributed 
simulation-based  collective  training. 

d.  Hardware/software  Support.  Engineering  support  is  critical,  particularly  on  the 
receiving  end  of  distributed  simulation.  If  a  group  of  students  is  convened  at  a  local 
training  site  but  unable  to  proceed  with  exercises  because  of  hook-up  or  other  equipment 
malfunctions,  considerable  time  and  money  will  be  wasted. 
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e.  Compatibility  with  Simulation  Upgrades  at  the  Emmitsburg  campus.  What  are 
the  implications  for  distributed  simulation  of  implementing  upgrades  at  Emmitsburg, 
beyond  the  initial  upgrade? 

f.  Training  System  Evaluation.  What  evidence  can  be  collected  concerning  the 
value  of  the  distributed  training  and  what  needs  to  be  improved  in  training  delivery? 

Recommendation:  To  address  networking  needs  systematically 
we  recommend  that  NFA  prepare  and  document  a  matrix  which  defines 
instructional  purposes,  goals,  conditions,  circumstances  along  one  axis, 
networking  methods  along  a  second  axis,  and  cost/  effectiveness  issues  along  a 
third  axis.  Much  useful  discussion  has  taken  place  at  NFA  about  elements  of  this 
matrix.  It  would  be  useful,  at  this  point,  to  begin  compiling  the  corporate  wisdom 
and  enhance  it  with  additional  information. 

’Long-Haul’  Networking  to  Other  EM  Agencies  and  Agencies  With  EM 
Missions 


Most  of  the  preceding  issues  and  recommendations  apply  here.  But  a  particular 
’lesson’  from  MDT2  may  be  applicable. 

The  cardinal  premise  of  MDT2,  reinforced  by  research  data,  is  that  multi-service  training 
should  focus  on  multi-service  tasks.  Soldiers  should  be  well  trained  in  service-specific 
unit  tasks  before  coming  together  for  joint  exercises.  The  effectiveness  -  or  at  least  the 
efficiency  -  of  these  exercises  will  be  significantly  impaired  othenwise.  This  premise  has 
a  critical  impact  on  decisions  about  fidelity  and  interactivity.  For  example,  the  simulators 
used  in  MDT2  are  not  designed  to  train  basic  skills  in  tank  or  aircraft  operation. 

Recommendation:  We  recommend  that  NFA  determine  whether  the 
above  premise  is  valid  for  emergency  management  training  and  if  so  what  implications 
it  has  for  fidelity  decisions. 
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CONCLUDING  COMMENTS 


A  clear  need  has  been  documented  to  enhance  simulation-based  training  of 
incident  command  at  the  NFA.  The  present  study  reinforces  an  earlier  determination  by 
panels  of  fire-fighting  subject  matter  experts.  It  is  feasible  to  successfully  upgrade 
training  technology  if  the  NFA  proceeds  in  carefully  measured  incremental  steps  and 
maintains  a  training  system  perspective,  with  significantly  increasing  training  value  as 
the  primary  goal. 

Each  incremental  enhancement  should  be  assessed  minimally  for  its  contribution 
to  increased  efficiency  of  training  delivery.  One  result  of  the  present  study  was  a  table 
which  documents  the  many  complex  roles  required  of  instructors  in  managing  1C 
simulations.  The  table  provides  a  concise  checklist  to  help  plan  and  assess  each 
enhancement. 

But  enhancements  should  also  be  assessed  for  their  contributions  to  increased 
training  effectiveness  and  ultimately  increased  training  value.  At  each  stage  considerable 
thought  and  analysis  should  be  aimed  at  answering  the  following:  How  much  fidelilty  and 
interactivity  are  useful  for  effective  1C  training?  We  have  suggested  that  a  key  to 
answering  this  question  is  a  clear  understanding  of  the  1C  behavior  which  needs  to  be 
shaped  (e.g.,  critical  decisions)  and  the  chain  of  preceding  cues  which  needs  to  trigger 
this  behavior. 

Finally,  we  have  tried  to  identify  in  this  study  a  few  lessons  learned,  experiences, 
and  research  products  from  the  military  which  may  provide  some  assistance  to  the  NFA 
in  a  very  exciting  and  far  reaching  effort  to  revolutionize  the  delivery  of  emergency 
management  training. 
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END  NOTES 


1.  The  facilities  described  were  those  observed  in  January,  1994.  As  of  this  writing 
(11/1/94)  NFA  is  procuring  computer  technology  to  support  management  and 
conduct  of  multi-alarm  training  simulations.  The  procurement  addresses  problems 
raised  in  this  report. 

2.  This  table  is  a  revised  version  of  one  presented  at  the  Human  Factors  and  Ergonomics 
Society  38th  Annual  Meeting,  in  Nashville,  TN. 

3.  Army  training  development  experience  suggests  the  wisdom  of  taking  and 
evaluating  each  step  before  proceeding  to  the  next. 
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APPENDIX 


RATIONALE  FOR  NFA/ARI  PARTNERSHIP 
Federal/DOD  Dual  Use  Policy 

The  goal  of  Federal  policy  on  ’dual-use’  is  to  increase  efficiency  and  reduce  costs 
by  encouraging  agencies  to  share  resources  and  products.  (U.S.  Office  of  the  Vice 
President,  1993).  The  DOD  goal  on  ’dual-use’  is  consistent  with  Federal  policy  but  has 
a  more  specific  focus.  The  DOD  goal  is  primarily  to  maintain  defense  readiness  in 
expectation  of  declining  funding  for  development,  acquisition,  and  maintenance  of  military 
technology  (Deutch,  1994).  DOD  strategy  is  to  seek  ’repositories’  for  military  technology 
in  the  civilian  sector  so  that  technology  will  be  available  or  recoverable  when  needed  for 
defense.  It’s  appropriate  to  apply  the  strategy  to  training  technology,  since  training  is 
critical  to  preparedness.  In  pursuing  the  strategy,  a  partnership  with  NFA  is  appropriate 
for  reasons  discussed  below. 

Common  Training  Development  Problems 

NFA  and  ARI  address  similar  work  environments  and  training  development 
problems;  high  pressure,  dangerous  missions  with  an  ’enemy’  to  defeat  by  ’combat’  and 
’support’  units.  Accordingly,  the  problems  of  fire-ground  and  military  commanders  are 
similar.  Both  have  to  manage  resources  to  defeat  a  deadly  enemy.  Both  have  to  develop 
goals,  strategies,  and  tactics,  but  require  expert  content  knowledge  as  well.  Finally,  both 
operate  under  time  stress  that  does  not  allow  exhaustive,  decision-tree  analysis  of 
alternative  tactics.  Training  for  both  environments  requires  effective  use  of  simulation 
and  post-incident  analysis  to  provide  transferable  and  generalizable  ’job’  experience. 

Transferable  ARI/Army  Experience  and  Products 

Tactical  Engagement  Simulation  (TES).  In  the  mid  1970s  the  US  Army,  with 
support  from  the  ARI  pioneered  the  development  and  use  of  two-sided,  free  play  training 
exercises  with  realistic  casualty  assessment.  From  research  on  that  technology  emerged 
guidelines  for  providing  diagnostic  feedback  in  the  form  of  After  Action  Reviews  (Hiller, 
Hardy,  &  Melizza,  1984;  Scott,  1983,  1984).  Later,  ARI  extended  this  research  to 
command  and  control  training  (Downs,  Johnson,  &  Fallesen,  1987;  Kaplan  &  Fallesen, 
1 986).  The  indispensable  contribution  of  feed  back  to  unit  training  effectiveness  was  also 
established  by  ARI  research  (Thomas  et  al,  1989).  The  AAR  methodology  and  its 
validation  are  relevant  to  1C  training  at  the  NFA. 

Decision  Making.  For  at  least  a  decade,  the  US  Army  Research  Institute  has 
conducted  research  on  simulation-based  command  and  control  of  unit  operations 
(Fallensen,  1993).  This  work  has  resulted  in  training  lessons  learned  (Kaplan,  1987)  and 
specific  decision  making  models  which  are  relevant  to  command  and  control  training  for 
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emergency  managements  (Cohen,  1994;  Halpin,  1993;  Klein,  Calderwood,  &  Clinton- 
cirocco,  1988,  Klein,  Zsambok,  &  Thordsen,  1993).  One  of  these  models  (Klein’s 
Recognition  Primed  Decision-Making)  has,  in  fact,  been  adopted  by  NFA  for  wide-spread 
use  in  its  1C  courses. 

Distributed  Training.  Distance  learning  and  more  recently  distributed  interactive 
simulation  research  by  ARI  can  provide  lessons  learned  and  research  products  to 
support  the  NFA  goal  of  distributing  1C  simulation  nation-wide.  (Bell,  Mastaglio,  and 
Moses  1993;  Boldovici  &  Bessemer,  1994;  Burnside,  1999;  Burnside  &  Bessemer,  1991; 
Mirabella,  Macpherson,  &  Patterson,  1989;  Hahn,  Harbour,  Wells,  Schurman,  &  Daveline, 
1990;  Harbour,  Daveline,  Keith,  Wells,  Shurman,  &  Hahn,  1990;  Hiller,  1988;  Phelps, 
Ashworth,  &  Hahn,  1991;  Linville,  Liebhaber,  &  Obermayer,  1991).  The  completed  and 
emerging  results  of  this  research  can  contribute  to  NFA’s  longer-range  objective  of 
exporting  training  through  ’long-haul’  technology.  For  example,  training  assessment 
methodology  developed  as  part  of  the  Multi-Service  Distributed  Test  Bed  (MDT2) 
demonstration  is  relevant  to  NFA  interests  in  training  simulation  system  assessment 
(Mirabella,  in  press). 

Army’s  Growing  Role  in  Peacetime  Emergency  Management. 

The  Army’s  growing  role  in  peacetime  fire  fighting  operations,  has  been  well 
publicized  (e.g..  Constant  &  Fink,  1994).  But  this  publicity  reflects  only  ’the  tip  of  the 
iceberg’.  A  wide  variety  of  Operations  Other  Than  War  (OOTW)  are  becoming 
increasingly  important  sources  of  missions  for  the  military.  For  example,  the  Military 
Intelligence  community  is  providing  critical  support  to  civilian  law  enforcement  in  drug 
interdiction  (Lamberson,  1995;  Santiago,  1995;  Schaubelt,  1995).  Disaster  relief  and 
peace  keeping  are  other  emergency  management  OOTWs  (Hasenauer,  1994a,  1994b, 
1994c,  1994d;  Ristau,  1994;  Yantis,  1994;  Ide,  1994a,  1994b;  Phoebus,  1994;  Lane, 
1994a,  1994b;  Kirchmann,  1994;  Harding,  1994;  Miles,  1994a,  1994b). 

With  these  added  missions  come  new  or  increasing  training  requirements.  Given 
the  anticipated  DOD  draw-downs  and  budget  cut-backs,  FEMA  can  become  a  significant 
alternative  source  of  some  of  that  training.  There  is  precedent  for  this,  since  NFA 
currently  trains  some  military  officers  at  its  Emmitsburg  facility.  With  increasing  capability 
through  simulation  upgrades  and  especially  distributed  simulation,  FEMA’s  role  in  training 
the  military  for  EM  could  expand  in  some  very  important  ways.  For  example,  where  rapid 
train-up  for  a  regional  fire  catastrophe  is  required  of  military  units,  ’long-haul’  simulation 
facilities  operated  by  NFA  could  be  very  responsive.  Given  the  possibility  of  such  a 
scenario,  an  NFA/ARI  partnership  which  contributes  to  long-range  NFA  goals  is  mutually 
advantageous. 
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